Short wave infrared image sensor with automatic exposure and dynamic range control

ABSTRACT

A system comprises an image sensor comprising an array of infrared sensors operable to capture an image of a scene, an integration timing circuit variably controlling an integration time of the infrared sensors, a readout integrated circuit (ROIC) operable to generate signals from the infrared sensors corresponding to the captured image of the scene, and an automatic gain control operable to applied a gain to the generated signals. The system further comprises a dynamic range compensation controller operable to receive image parameters corresponding to the generated signals, compare the received image parameters to target image parameters, and generate an integration timing signal for controlling the integration timing circuit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 62/588,879 filed Nov. 20, 2017 and entitled“SHORT WAVE INFRARED IMAGE SENSOR WITH AUTOMATIC EXPOSURE AND DYNAMICRANGE CONTROL,” which is incorporated herein by reference in itsentirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

This invention was made in part with government support under contract#FA8650-14-C-5508, awarded by the United States Air Force. The UnitedStates Government may have certain rights in and to this invention.

TECHNICAL FIELD

One or more embodiments of the present disclosure relate generally toinfrared imaging devices and more particularly, for example, tooptimization of signal to noise performance in short wave infraredimaging devices.

BACKGROUND

There are a wide variety of image detectors, such as visible imagedetectors, infrared image detectors, or other types of image detectorsthat may be provided in an image detector array for capturing an image.As an example, a plurality of sensors may be provided in an imagedetector array to detect electromagnetic radiation at desiredwavelengths. Such detectors may be connected to or part of unit cells ina read out integrated circuit (ROIC) which capture image data inresponse to the detected radiation and then communicate that image datato external electronics. The combination of a detector array with anROIC is known either as a focal plane array (FPA) or an image sensor.Advances in process technology for FPAs and image processing have led toincreased capabilities and sophistication of resulting imaging systems.Many imaging applications such as short wave infrared (SWIR) facechallenges when imaging scenes having a large dynamic range. A SWIRcamera, for example, may operate from low light conditions with a photonflux of mid 10¹¹ (ph/cm²s) up to daylight conditions of mid 10¹⁵(ph/cm²s). There is a continued need in the art for more efficient,accurate and higher quality imaging systems for use across a range ofimaging environments.

SUMMARY

Various techniques are provided for implementing a short wave infraredimage sensor with automatic exposure and dynamic range control. Invarious embodiments, a system comprises an image sensor and a dynamicrange compensation controller. The image sensor may comprise an array ofinfrared sensors (e.g., a short wave infrared image sensor) operable tocapture an image of a scene, an integration timing circuit variablycontrolling an integration time of the infrared sensors, a readoutintegrated circuit (ROIC) operable to generate signals from the infraredsensors corresponding to the captured image of the scene, and anautomatic gain controller operable to apply a gain to the generatedsignals. The dynamic range compensation controller may be operable toreceive image characteristics corresponding to the generated signals,compare the received image characteristics to target imagecharacteristics, and adjust an integration timing signal for controllingthe integration timing circuit to align the received imagecharacteristics with the target image characteristics.

In some embodiments, the image characteristics comprise a received imagehistogram and a target image histogram and the dynamic rangecompensation controller is operable to adjust the integration timingsignal to align the received image histogram to the target imagehistogram. In one embodiment, the dynamic range compensation controllerincludes a peak mode wherein the received image histogram is adjusted toalign a top of the received image histogram with a top of the targethistogram. In another embodiment, the dynamic range compensationcontroller includes an average mode wherein the received image histogramis adjusted to align a mean of the received image histogram with a meanof the target histogram. The dynamic range compensation controller mayalso comprise an automatic exposure control (AEC) module operable toadjust the integration timing signal, and a gain control module operableto generate a gain control signal for controlling the automatic gaincontroller. The AEC module may be operable to detect whether theintegration timing signal is equal to or below a minimum integrationthreshold, and wherein the gain control module is operable to adjust thegain control signal to an available lower gain mode if the imagecharacteristics and the target image characteristic are not aligned, anddetect whether the integration timing signal is equal to or above amaximum integration threshold, and wherein the gain control module isoperable to adjust the gain control signal to a higher gain mode whenthe image characteristics and the target image characteristic are notaligned.

In various embodiments, the dynamic range compensation controller isfurther operable to calculate a step size based on the comparisonbetween the received image characteristics and the target imagecharacteristics and wherein the integration timing signal is adjusted byapplying the step size to a current integration timing signal. Thedynamic range compensation controller may further a state machineoperable to perform the automatic exposure control.

In various embodiments, a method comprises capturing an image of a sceneusing an array of infrared sensors (e.g., short wave infrared imagesensors), variably controlling an integration time of the infraredsensors using an integration timing circuit, generating signals from theinfrared sensors corresponding to the captured image of the scene usinga readout integrated circuit (ROIC), applying a gain to the generatedsignals, using an automatic gain controller; and compensating for adynamic range of the image using a dynamic range controller, includingreceiving image characteristics corresponding to the generated signals,comparing the received image characteristics to target imagecharacteristics, and generating an integration timing signal forcontrolling the integration timing.

In some embodiments, the image characteristics may comprise a receivedimage histogram and a target image histogram and the integration timingsignal is adjusted to align the received image histogram to the targetimage histogram. In various embodiments, the adjustment of theintegration timing signal may comprise operating in a peak mode whereinthe received image histogram is adjusted to align a top of the receivedimage histogram with a top of the target histogram, or an average modewherein the received image histogram is adjusted to align a mean of thereceived image histogram with a mean of the target histogram. Theintegration timing signal may be adjusted using an automatic exposurecontrol (AEC) module, and a gain control signal may be generated forcontrolling the automatic gain controller. The integration timing signalmay be adjusted by generating a step size based on the comparisonbetween the received image characteristics and the target imagecharacteristics and adjusting the integration timing signal by applyingthe step size to a current integration timing signal.

The method may further comprise detecting whether the integration timingsignal is equal to or below a minimum integration threshold, andadjusting the gain control signal to an available lower gain mode if thereceived image characteristics and the target image characteristic arenot aligned, and detecting whether the integration timing signal isequal to or above a maximum integration threshold, and adjusting thegain control signal to a higher gain mode when the image characteristicsand the target image characteristic are not aligned.

The scope of the invention is defined by the claims, which areincorporated into this section by reference. A more completeunderstanding of embodiments of the invention will be afforded to thoseskilled in the art, as well as a realization of additional advantagesthereof, by a consideration of the following detailed description of oneor more embodiments. Reference will be made to the appended sheets ofdrawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example imaging system in accordance with anembodiment of the disclosure.

FIG. 2 illustrates a block diagram of an infrared sensor assemblyincluding an array of infrared sensors in accordance with an embodimentof the disclosure.

FIG. 3 is a block diagram of an infrared imaging device in accordancewith an embodiment of the disclosure.

FIG. 4 illustrates an example scenario where an image histogramapproaches an image target in accordance with an embodiment of thedisclosure.

FIG. 5 illustrates an example of a state machine for the operation of adynamic range controller in accordance with an embodiment of thedisclosure.

FIG. 6 illustrates an approach of the current integration time to thetarget value in accordance with an embodiment of the disclosure.

FIG. 7 illustrates an embodiment depicting a scenario where an imagehistogram approaches an image target in accordance with an embodiment ofthe disclosure.

FIGS. 8A-B illustrate an example flow of control of the automaticexposure control in accordance with an embodiment of the disclosure.

FIGS. 9A-D illustrate example operation of an AEC module in accordancewith embodiments of the disclosure.

FIG. 10 is a flow diagram illustrating an example process for operatingan image sensor in accordance with embodiments of the disclosure.

Embodiments of the disclosure and their advantages are best understoodby referring to the detailed description that follows. It should beappreciated that like reference numerals are used to identify likeelements illustrated in one or more of the figures.

DETAILED DESCRIPTION

Various embodiments for implementing a short wave infrared image sensorwith automatic exposure and dynamic range control are disclosed herein.Referring to FIG. 1, a block diagram of a system (e.g., an imagingsystem such as an infrared camera) for capturing and processing imagesin accordance with one or more embodiments will now be described. System100 comprises, in one implementation, a processing component 110, amemory component 120, an image capture component 130, a controlcomponent 140, and/or a display component 150. System 100 may furtherinclude a sensing component 160.

System 100 may represent, for example, an imaging system such as aninfrared imaging device, or a multi-band imaging device for capturingand processing images, such as video images of a scene 170. In someembodiments, system 100 may represent any type of infrared cameraadapted to detect infrared radiation and provide representative data andinformation (e.g., infrared image data of a scene) or may represent moregenerally any type of electro-optical sensor system. As examples, system100 may represent an infrared camera, a dual band imager such as a nightvision imager that operates to sense reflected visible and/or short-waveinfrared (SWIR) light for high resolution images and long-wave infrared(LWIR) radiation for thermal imaging, or an imager for sensing bothshort wave and long wave radiation simultaneously for providingindependent image information. System 100 may comprise a portable deviceand may be incorporated, e.g., into a vehicle (e.g., hand-held devices,an automobile or other type of land-based vehicle, an aircraft, a marinecraft, or a spacecraft) or a non-mobile installation requiring infraredimages to be stored and/or displayed or may comprise a distributednetworked system.

In various embodiments, processing component 110 may comprise any typeof a processor or a logic device (e.g., a programmable logic device(PLD) configured to perform processing functions). Processing component110 may be adapted to interface and communicate with components 120,130, 140, and 150 to perform method and processing steps and/oroperations, as described herein such as controlling biasing and otherfunctions (e.g., values for elements such as variable resistors andcurrent sources, switch settings for timing such as for switchedcapacitor filters, ramp voltage values, etc.) along with conventionalsystem processing functions as would be understood by one skilled in theart.

Memory component 120 comprises, in one embodiment, one or more memorydevices adapted to store data and information, including for exampleinfrared data and information. Memory device 120 may comprise one ormore various types of memory devices including volatile and non-volatilememory devices. Processing component 110 may be adapted to executesoftware stored in memory component 120 so as to perform method andprocess steps and/or operations described herein.

Image capture component 130 comprises, in one embodiment, any type ofimage sensor, such as, for example, an image sensor having one or moreimage detector elements such as infrared photodetector elements (e.g.,any type of multi-pixel infrared detector, such as a focal plane arrayas described hereinafter) for capturing infrared image data (e.g., stillimage data and/or video data) representative of an scene such as scene170. If desired, image capture component 130 may include one or morearrays of other detector elements such as uncooled detector elements(e.g., uncooled microbolometer sensors), cooled detector elements (e.g.,detector elements such as photovoltaic or quantum structure elementsthat are cooled using a cryogen coupled to the array or using arefrigeration system), InSb detector elements, quantum structuredetector elements, InGaAs detector elements, or other types of sensors.

In one implementation, image capture component 130 may be configured togenerate digital image data representing incoming image light from scene170. Image capture component 130 may include one or more signalprocessing components such as analog-to-digital converters included aspart of an infrared sensor or separate from the infrared sensor as partof system 100. In one aspect, infrared image data (e.g., infrared videodata) may comprise non-uniform data (e.g., real image data) of a scenesuch as scene 170. Processing component 110 may be adapted to processthe infrared image data (e.g., to provide processed image data), storethe infrared image data in memory component 120, and/or retrieve storedinfrared image data from memory component 120. For example, processingcomponent 110 may be adapted to process infrared image data stored inmemory component 120 to provide processed image data and information(e.g., captured and/or processed infrared image data).

Control component 140 comprises, in one embodiment, a user input and/orinterface device. For example, the user input and/or interface devicemay represent a rotatable knob (e.g., potentiometer), push buttons,slide bar, keyboard, etc., that is adapted to generate a user inputcontrol signal. Processing component 110 may be adapted to sense controlinput signals from a user via control component 140 and respond to anysensed control input signals received therefrom. Processing component110 may be adapted to interpret such a control input signal as aparameter value, as generally understood by one skilled in the art.

In one embodiment, control component 140 may comprise a control unit(e.g., a wired or wireless handheld control unit) having push buttonsadapted to interface with a user and receive user input control values.In one implementation, the push buttons of the control unit may be usedto control various functions of the system 100, such as autofocus, menuenable and selection, field of view, brightness, contrast, noisefiltering, high pass filtering, low pass filtering, and/or various otherfeatures as understood by one skilled in the art.

In one embodiment, control component 140 may optionally includetemperature control components for cooling or heating an image sensor.Temperature control components may include a container such as a Dewarcontaining a cryogenic liquid and a thermally conductive coupling membercoupled between the cryogenic liquid and a sensor structure on which anarray of detectors is formed. However, this is merely illustrative. Ifdesired, image capture component 130 may be an uncooled image capturecomponent.

Display component 150 comprises, in one embodiment, an image displaydevice (e.g., a liquid crystal display (LCD) or various other types ofgenerally known video displays or monitors). Processing component 110may be adapted to display image data and information on the displaycomponent 150. Processing component 110 may be adapted to retrieve imagedata and information from memory component 120 and display any retrievedimage data and information on display component 150. Display component150 may comprise display electronics, which may be utilized byprocessing component 110 to display image data and information (e.g.,infrared images). Display component 150 may be adapted to receive imagedata and information directly from image capture component 130 via theprocessing component 110, or the image data and information may betransferred from memory component 120 via processing component 110.

Sensing component 160 comprises, in one embodiment, one or more sensorsof various types, depending on the application or implementationrequirements, as would be understood by one skilled in the art. Thesensors of optional sensing component 160 provide data and/orinformation to at least processing component 110. In one aspect,processing component 110 may be adapted to communicate with sensingcomponent 160 (e.g., by receiving sensor information from sensingcomponent 160) and with image capture component 130 (e.g., by receivingdata and information from image capture component 130 and providingand/or receiving command, control, and/or other information to and/orfrom one or more other components of system 100).

In various implementations, sensing component 160 may provideinformation regarding environmental conditions, such as outsidetemperature, lighting conditions (e.g., day, night, dusk, and/or dawn),humidity level, specific weather conditions (e.g., sun, rain, and/orsnow), distance (e.g., laser rangefinder), and/or whether a tunnel orother type of enclosure has been entered or exited. Sensing component160 may represent conventional sensors as generally known by one skilledin the art for monitoring various conditions (e.g., environmentalconditions) that may have an effect (e.g., on the image appearance) onthe data provided by image capture component 130.

In some implementations, optional sensing component 160 (e.g., one ormore of sensors) may comprise devices that relay information toprocessing component 110 via wired and/or wireless communication. Forexample, optional sensing component 160 may be adapted to receiveinformation from a satellite, through a local broadcast (e.g., radiofrequency (RF)) transmission, through a mobile or cellular networkand/or through information beacons in an infrastructure (e.g., atransportation or highway information beacon infrastructure), or variousother wired and/or wireless techniques.

In various embodiments, components of system 100 may be combined and/orimplemented or not, as desired or depending on the application orrequirements, with system 100 representing various functional blocks ofa related system. In one example, processing component 110 may becombined with memory component 120, image capture component 130, displaycomponent 150, and/or optional sensing component 160. In anotherexample, processing component 110 may be combined with image capturecomponent 130 with only certain functions of processing component 110performed by circuitry (e.g., a processor, a microprocessor, a logicdevice, a microcontroller, etc.) within image capture component 130.Furthermore, various components of system 100 may be remote from eachother (e.g., image capture component 130 may comprise a remote sensorwith processing component 110, etc. representing a computer that may ormay not be in communication with image capture component 130).

FIG. 2 is a block diagram of an infrared sensor assembly 200 accordancewith an embodiment of the invention. The infrared sensor assembly 200may be a focal plane array, for example, implemented as an image sensorin image capture component 130 of FIG. 1.

In the illustrated embodiment, the infrared sensor assembly 200 includesan array of infrared sensors 205 provided as part of a unit cell arrayof a ROIC 202. The ROIC 202 includes bias generation and timing controlcircuitry 240, column amplifiers 232, a column multiplexer 230, a rowmultiplexer 220, and an output amplifier 250. Image frames (e.g.,thermal images) captured by infrared sensors 205 may be provided byoutput amplifier 250 to processing component 110 and/or any otherappropriate components to perform various processing techniquesdescribed herein. Although an 8 by 8 array is shown in FIG. 2, anydesired array configuration may be used in other embodiments.

The infrared sensor assembly 200 may capture images (e.g., image frames)and provide such images from its ROIC 202 at various rates. In someembodiments, each unit cell 210 may be configured to integrate andreadout image signals generated by detectors in multiple detector rows.In this type of configuration, a single unit cell 210 may be used tointegrate charges, during multiple integration times, from multipledetectors, including detectors associated with other unit cells 210. Forexample, a unit cell 210 in a first row may be used to integrate imagecharges from its associated detector and from one or more detectors inadjacent detector rows.

Processing component 110 may be used to perform appropriate processingof captured infrared images and may be implemented in accordance withany appropriate architecture. In one embodiment, processing component110 may be implemented as an ASIC. In this regard, such an ASIC may beconfigured to perform image processing with high performance and/or highefficiency. In another embodiment, processing component 110 may beimplemented with a general purpose central processing unit (CPU) whichmay be configured to execute appropriate software instructions toperform image processing, coordinate and perform image processing withvarious image processing blocks, coordinate interfacing betweenprocessing component 110 and a host device, and/or other operations. Inyet another embodiment, processing component 110 may be implemented witha field programmable gate array (FPGA). Processing module 110 may beimplemented with other types of processing and/or logic circuits inother embodiments as would be understood by one skilled in the art.

In these and other embodiments, processing component 110 may also beimplemented with other components where appropriate, such as, volatilememory, non-volatile memory, and/or one or more interfaces (e.g.,infrared detector interfaces, inter-integrated circuit (I2C) interfaces,mobile industry processor interfaces (MIPI), joint test action group(JTAG) interfaces (e.g., IEEE 1149.1 standard test access port andboundary-scan architecture), and/or other interfaces).

One or more control circuits may be provided as part of and/or separatefrom infrared sensor assembly 200 to provide various signals furtherdescribed herein. Such control circuits may be implemented in accordancewith any appropriate control circuits such as one or more processors(e.g., processing components 110), logic, clocks, and/or other circuitsas may be desired for particular implementations. In variousembodiments, the components of the system100 and/or infrared imagingmodule 200 may be implemented as a local or distributed system withcomponents in communication with each other over wired and/or wirelessnetworks. Accordingly, the various operations identified in thisdisclosure may be performed by local and/or remote components as may bedesired in particular implementations.

Referring to FIG. 3, an infrared image device 300, such as a SWIRcamera, is illustrated in accordance with an embodiment of the presentdisclosure. Imaging device 300 includes a lens 305 focusing light onto aunit cell array 310 of an image sensor 312. It will be appreciated thatimage sensor 312 need not be limited to a SWIR image sensor but maycomprise a visible light CMOS sensor or other suitable types of imagesensors. An integration timing circuit 320 controls the integration ofthe light received by the image sensor 312 to generate an analog imagesignal. The analog image signal is gain adjusted through an automaticgain control (AGC) unit 325 and digitized in an analog-to-digitalconverter (ADC) 330. An image processor 340 processes the resulting rawdigital signal from ADC 330 to produce the desired digitized imagesignal.

In various embodiments, during daylight operation, image sensor 312 mayreceive sufficient light such that AGC 325 need not activate. Automaticgain control 325 controls the gain applied to an input signal based uponfeedback from the image processor 340 and dynamic range compensationcontroller 350. For example, each pixel in the digitized image fromimage processor 340 has some dynamic range—for example, if the dynamicrange is 8 bits, each pixel value can range from zero to 255. The imagewould be completely saturated if each 8-bit pixel had a value of 255 andwould be totally dark if each pixel had a value of zero. Thus, imageprocessor 340 may provide a feedback reference signal to the dynamicrange compensation controller 350 that represents an average pixelvalue. In various embodiments, the automatic gain control may beperformed in the analog domain or in the digital domain.

In one mode of operation, during daylight the average pixel valuereceived from the image processor 340 is provided to the AGC unit 325,which provides only a baseline amount of gain (such as through avariable gain amplifier). As daylight fades, the average pixel valuedrops below a desired reference value, whereupon a control signal isgenerated to increase the gain applied to input image signal.Conversely, as light intensity increases, the control signal woulddecrease the amount of gain applied to input signal.

It will be appreciated that automatic gain control could also occur inthe digital domain. For example, a digital gain may be applied via ahistogram stretching or other suitable technique on the digitized image.Histogram stretching may be applied, for example, to an underexposedimage that would result from low-light conditions. The relatively weakimage signal in such a case would occupy the lower part of the totaldynamic range. For example, consider an example embodiment in which eachimage pixel has a dynamic range of 8 bits such that any given pixel hasa digital value ranging from zero to 255. In a low-light conditionwithout analog AGC, the relatively weak image signal may be such that ahistogram for the pixel values shows that most of the pixels areconcentrated in a low range such as from zero to 50. Histogramstretching may then be applied to increase image quality by betteroccupying the available dynamic range. The amount of histogramstretching that is applied may thus be considered as a digital gain.Regardless of where the gain is controlled, the amount of gain appliedis responsive to some sort of control signal. The control signal thusserves as a proxy for determining whether external lighting conditionsare such that the image sensor should be cooled.

Dynamic Range Control

The dynamic range compensation controller 350 includes an automaticexposure module 352 providing automatic exposure control (AEC) and again control module 354 that control the dynamic range using feedbackfrom the image processor 340 to automatically control the sensorintegration time and gain mode to optimize image signal to noiseperformance. In one embodiment, the infrared image device 300 is a SWIRcamera and the unit cell array comprises an InGaAs detector. Imaging inthe SWIR band is often challenging where scenes have large dynamicrange. In a common mode of operation, a SWIR camera may operate from lowlight conditions with a photon flux of mid 1011(ph/cm2s) up to daylightconditions of mid 1015(ph/cm2s). The InGaAs detector can convert thisphoton flux to a current over the entire range, and this current issampled and processed into a digital signal at the output of the imagesensor 312 (e.g., a focal plane array integrated circuit). To provide asignal with good signal to noise characteristics, the readout integratedcircuit of the image sensor 312 has a variable integration time,controlled through integration timing circuit 320, which is analogous toa shutter time. The illustrated embodiment also provides multiple gainmodes (e.g., 3 gain modes) through the automatic gain control unit 325,which are analogous to a camera film speed. In various embodiments, whenunder low light conditions the integration time may be long and the gainmay be set to high mode for best performance, and when the sceneillumination is high (daylight) the integration time may be short andthe gain may be set in a low mode.

In operation, the automatic exposure module 352 controls the integrationtime of the image sensor 312. In one embodiment, statistics of thecurrent image of the image processor are measured and, based on the meanof the peak level of the image histogram distribution, the integrationtime is adjusted. The control loop operates continuously during AECoperation. As the scene gets darker the integration time is increased,or as the scene gets brighter the integration time is decreased. Thecontrol loop may have user inputs which control the image distributionsetpoints, and the rate at which the algorithm approaches the setpoint.For example, user input options may include selection of a gain table,image information (e.g., mean image, image peak and image based values),integration information (e.g., target values), enable/disable of autoexposure and gain switch modes, integration time, gain mode settings,auto exposure settings and other user input options.

The gain control module 354 of the dynamic range compensation controller350 controls the current gain state. In one embodiment, three gainstates are available: high, medium and low. Each gain state has anoptimum range of operation. Under low light conditions the AEC functionwill increase the integration time to the maximum attempting to achievethe defined operation point. If the integration time is maximized andthe image histogram is still in the lower part of the range, the gaincontrol module 354 will switch the gain state to a higher gain. Underbright conditions the automatic exposure module 352 will decrease theintegration time to a minimum period attempting to achieve the definedoperation point. If the integration time is minimized and a portion ofthe scene is still saturated, the gain control module 354 will switchthe gain state to a lower gain.

Referring to FIG. 4, an embodiment depicting an automatic exposurecontrol scenario where the image histogram approaches the target fromthe left is illustrated. The AEC module 352 may run in a peak-mode or anaverage-mode in the illustrated embodiment. In peak mode, the AEC module352 attempts to align the top of the histogram (the top-most bin inwhich pixels are populated) to the target value. The target value may bespecified as a percentage of the maximum range. Bins may be measured inunits of max-count/8, or 16383/8, for a total parameter range of 0 to2047. In illustrated diagrams, the target for peak-mode is set to 80%,and given a target of 80% (i.e., 80% of 2047 is a target of 1638), thealgorithm attempts to align the top-most bin populated with pixels to1638.

In various embodiments, the AEC module 352 operates on a histogram whichhas been modified to remove outliers at the top and bottom of thedistribution. For example, a head clip function may remove a percentageof the pixels at the top of the histogram, which are excluded from theAEC calculations, and a tail clip function may remove a percentage ofthe pixels at the bottom of the histogram, which are excluded from theAEC calculations. The gain control module 354 becomes active when theAEC module cannot align the histogram, and parameters are set to preventexcessive switching between gain modes (e.g., low, medium, high). Forexample, if the current scene parameter (image mean intensity or imagepeak intensity) is less than the target value, then the integration timeis increased to make the scene brighter. If the current scene parameteris greater than the target value, then the integration time is decreasedto make the scene darker.

In one embodiment, the adjustment in integration time is made bydetermining a stepsize and applying the step size to the currentintegration time. For example, the next step size may be calculated as amaximum step size (e.g., maximum percentage change of intensity) timesthe difference between the target value and the current imageparameters. The step size is then applied to the current integrationtime. The new integration time is then provided to the integrationtiming circuit to control the integration time of the sensor array.

As shown in FIG. 4, since the next step is calculated to be 20% of thedifference between current and target, for a stationary target, eachstep size becomes progressively smaller. In one embodiment, thisasymptotic approach is complicated by the fact that in order to ensurethat no undesirable artefacts are introduced into the video, the imagesensor controller does not allow arbitrary values to be written into theintegration time register of the image sensor. That is, the AEC modulecalculates a target integration time and requests the change by callingthe image sensor controller. But, image sensor controller enforces aneffective granularity on the actual setting it makes to this parameter.If the AEC module's calculated step-size becomes less than thisgranularity, the image sensor controller rounds the integration timedown to a setting that conforms to the granularity. Under thesecircumstances, from the AEC Module's perspective, the net effect is thatthis leaves the integration time unchanged. Unless something is doneabout this, the algorithm will continue to try to move the next steptowards the target, fail, retry, fail, etc. In order to overcome thislimitation, a state machine is maintained to ensure that the scenestatistics converge to the desired target.

An embodiment of an AEC state machine 500 in accordance with the presentdisclosure is illustrated in FIG. 5. In the illustrated embodiment, Modecan be set to either Peak or Average Mode, and the DEAD_BAND is set toapproximately +/−100 bin/8, or approximately +/−900-1000 counts. Theinitial state is the normal step state 510 during which the AEC moduleadjusts the integration timing to move the image histograms towards atarget. When the calculated stepsize is less or equal to an integrationtime granularity value, machine enters the line step state 520. Thegranularity of allowed integration time may depend on an operating modeof the SWIR device, such as whether the device is currently doingintegrate then read or integrate while read. When the pre-integrationtime is determined to be greater than or equal to the readout time, thenthe granularity is effectively turned off by setting it to a“granularity” of one clock. But when the camera is in IWR mode(pre-integration time is determined to be less than the readout time),then the granularity is set to the number of clocks per line (e.g., 180clocks). If the target is reach, then the machine enters the no stepstate 530 until the calculated step size is greater than or equal to theDEAD_BAND value.

An illustration of the approach of the current integration time to thetarget value is shown in FIG. 6. If the calculated step-size becomesless than the current image sensor integration time granularity, thenthe integration time is adjusted by the current image sensor-allowedintegration time granularity, until the target value is reached. Thetarget value is reached because the approach in this state is no longerasymptotic. Generally, the algorithm doesn't hit the exact target value.

Referring to FIG. 7, when the current value approaches the target valuefrom the left, the algorithm considers the target “met” when the currentvalue crosses the target value (becomes greater than or equal to thetarget value). When the current value approaches the target value fromthe right, in order to ensure that the algorithm deterministically landsat the same spot for a given target value, when the current valuecrosses the target value (becomes less than), the algorithm reverses byone step. Finally, in order to eliminate scene flicker, the algorithmwill not adjust the scene any further until the calculated step-sizebecomes greater than the dead-band (approximately +/−900-1000 counts).

FIGS. 8A-B illustrates an example flow control of the automatic exposurecontrol in accordance with an embodiment of the disclosure. As used inthe drawings, a solid line indicates a function call, with the arrowheadindicating the direction of the call, and a dashed line indicates thereturn from a function call. A return value is optionally indicatedwhere the return value is important to the information being conveyed.Initialization is called, which initializes the AEC module and loads itsinternal parameters with default values. After the imaging device (e.g.,SWIR camera) has been initialized and begins processing video, theexecution of the AEC model is performed from an IdleTask logic functionas shown in diagram. The auto exposure module determines the currentscene parameters and target scene parameters (autoexp_run( )) andcalculates a stepsize and direction of the next step based onprogrammable AEC parameters. The integration time is updatedaccordingly.

In various embodiments, the AEC module is controlled by parameters whichmay include:

-   -   State: Keeps track of whether the AEC Module is currently        disabled or enabled.    -   Mode: Keeps track of whether the AEC Module is currently        operating in “peak” mode or “average” mode. Note that when time        the mode is changed, the internal AEC State Machine is reset to        the NORMAL_STEP_STATE.    -   PercentOfMaxRange: This parameter determines the percentage of        the maximum range that determines the target value that the        algorithm will shoot for. Note that each mode has its own unique        parameter setting.    -   MaxPercentChange: This parameter determines the maximum stepsize        (in percent) that the algorithm will move the current scene        towards the target. Note that each mode has its own unique        parameter setting.    -   initialized: This flag allows the algorithm to prevent any        internal processing to be performed until the AUTOEXP_Init( )        function has been called.

Gain switch control module parameters are also used in the AEC moduleoperation. In various embodiments, the AEC module is controlled byparameters which may include:

-   -   AECEnable, If AEC is not enabled, then Gain Switch is not        active.    -   GainSwitchEnable, Gain switch can be enabled/disabled, but it        will only be active if AEC is also enabled.    -   CurrentGainState. This variable probably already exists        somewhere else.    -   Switch Up parameters: These parameters determine if sensor gain        will switch to a higher state (Low to Medium or Medium to high).        -   SwitchUpTint: An integration time input variable in units of            percent of frame time.        -   SwitchUpNow: A Boolean variable. If            IntegratedTint>SwitchUpTint then SwitchUpNow is true    -   Switch Down Parameters: These parameter determine if sensor gain        will switch to a lower state (High to Medium or Medium to high).        -   SwitchDownTint: An integration time input variable in units            of percent of frame time.        -   SwitchDownNow: A Boolean variable. If            IntegratedTint>SwitchDownTint then SwitchDownNow is true.    -   Relative gain state parameters: These parameters define the        relative sensor gains. Units are multiples of the low gain        value. (Typical values shown here are for the 1202 Version 1,        Option 2)        -   LowGainValue; Programmable with default value 1.4        -   MediumGainValue: Programmable with default value 17        -   HighGainValue: Programmable with default value 46    -   Integration Time Sampling:        -   IntegrateSamples: Units number of frames to integrate. The            integration time will be integrated (averaged) for this            number of frames.        -   IntegratedTint: The average Tint over IntegrateSamples of            Tint. The decision to change gain state will be based on            this value.    -   A/D offset: Units percent of the total digital range. The range        checking for switching from lower to higher gain states is based        on the ratio of relative gain states times the total digital        range. There is likely an offset between zero counts and minimum        level of the output. This parameter is used to offset the range        checking levels.

When the gain switch control module is operating in a high gain mode,then if the integration time is turned down because the light is brightor the dark current is high or the histogram target is low, and thehistogram is close to target or the histogram cannot get down to thetarget and the system has stabilized so the step size is small, thenswitch down.

When the gain switch control module is operating in a medium gain mode,then if the integration time is turned down because the light is brightor the dark current is high or the histogram target is low, and thehistogram is close to target or the histogram cannot get down to thetarget and the system has stabilized so the stepsize is small, thenswitch down.

When the gain switch control module is operating in a medium gain mode,then if the integration time is turned up because the light is dim andthe dark current is low and the histogram target is high, and the systemhas stabilized so the histogram is close to target or cannot get up tothe target, then switch up, unless the histogram bin is greater than ⅓of the range because high gain range is ⅓ of the medium gain range.

When the gain switch control module is operating in a low gain mode,then if the integration time is turned up because the light is dim andthe dark current is low and the histogram target is high, and the systemhas stabilized so the histogram is close to target or cannot get up tothe target, then switch up, unless the histogram bin is greater than1/12 of the range because medium gain range is 1/12 of the low gainrange.

The large difference between gain states may result in bright jumps whenthe gain switches up, and dark jumps when the gain switches down. Thejump will exist until the AEC function corrects the integration time. InAutomatic Gain Switch mode the expected integration time in the new gainstate can be predicted based on the gain ratio from the old gain state.A predictive tint adjustment function performs a onetime tint adjustmentdirectly after the gain switch, then lets the AEC module complete theintegration time tuning. In one embodiment, the new tint value equalsthe old tint value times a ratio between the current state and nextstate values.

In various embodiments, the image sensor produces a 14-bit output andhas several offsets. For example, the output of the FPA may have a rangefrom 3.0V at starvation to 1.0V at saturation. The saturation leveltends to be constant, but the starvation will vary slightly with gainstate, and skim voltage setting. The A/D converter is configured todigitize 3.2V, slightly more than the actual voltage swing. The centerof the A/D range is also connected to a programmable voltage nominallyset to 2.0V. There is also some dark current in the detector whichvaries with detector temperature. The result is the minimum countsoutput by the camera in the dark at minimum integration time is greaterthan zero. The table switch level checking test should use an offsetvariable to account for this fixed offset.

Embodiments of the operation of the AEC module will now be furtherdescribed with reference to FIGS. 9A-D. Traditional sensors relied on2-point non-uniformity correction to provide pixel normalized video.This mode has some limitations: (i) fixed integration period (exposureperiod) per non-uniformity correction (NUC) table; (ii) NUC tableaccounts for non-scene related artifacts such as ROIC offset per pixeland dark currents of the detector; and (iii) limited intrascene dynamicrange due to limitation imposed by (i) and limited well capacity of eachpixel unit cell.

Automatic Exposure Control (AEC) allows dynamic adjustment of intrascenerange while maintaining “acceptable” level of video fidelity. AEC altersthe integration period (exposure) to maintain intrascene end-to-endrange at a predefined range of the sensor's electrical output rangewithout switching NUC tables. A graphical representation of scenedynamic range with respect to sensor dynamic range is shown in FIG. 9A.The graph on the left side represents two opposite conditions; scenerange as collected by the sensor is less than ideal size (red) and thescene range is larger than the ideal size (in dotted blue). Increasingthe exposure control (for red) and decreasing exposure control (forblue) will eventually converge to the optimized condition as shown inthe right side graph.

Two parameters for AEC are the target value and step size; both inpercentage. Target value is in percentage of the sensor digital dynamicrange. In one embodiment, this is percentage of 16383 or 14 bits. Stepsize is the change in exposure period based on the error (peak totarget) times the step size. Using a “step size” multiplier ensures overor critically dampen convergence to the Target Value. The graph of FIG.9B shows simulation (blue) and actual data (brown) of the AEC responseto ˜50% reduction in scene brightness with “Step Size” set to 20%.

Response to an over-bright scene is shown in the graphs of FIG. 9C. Theleft most graph is steady state condition for the sensor. The middlegraph shows that some of the pixels are “seeing” something very brightand the new histogram graph is shown with the bright population shown inred line. With the new population distribution, Tail_Top has moved awayfrom the target value and is reporting much higher value (dotted redvertical line vs previous tail top shown in green dotted vertical line).In response to new distribution, AEC will initiate reducing the exposureperiod to bring Tail_Top to the Target value. The right most graph showsthe steady state condition for the new scene with over-bright pixels. Ascan be seen by the graph, the initial steady state population is nowmoved to the left, making these pixels much darker and in someconditions loses their video grey scale and most scene may be shown asjust “black”. However the new over-bright pixels will use many of thevideo grey scale so that the bright pixels will show some details.

The above example is peak-AEC where the top (peak value) of thehistogram bin is adjusted to equal the target value. Mean-AEC uses thearray mean to equate to the target; graphical representation is shownbelow. Similar response except that the mean value does not vary as muchas peak. In this mode, it will preserve the bulk of the grey scale tothe original population and as a result renders the over-bright pixelsas one bright shade of grey with no details. In either case, due todegradation in the sensor uniformity, the usable limit is from ½X to 2Xof the integration period used to generate the NUC coefficients (NUCtable).

Referring to FIG. 10, an example process 1000 for operating an imagesensor in accordance with embodiments of the disclosure will now bedescribed. In step 1010, infrared radiation received by an array ofinfrared sensors is integrated in accordance with a current integrationstiming setting to produce analog image signals. The analog image signalsare then gain adjusted in step 1020 using a current gain mode. The imageis processed in step 1030 to produce a desired image signal. Imagecharacteristics are calculated from the processed image signal and fedback to a dynamic range compensation controller for automatic exposureand gain control. In step 1040, the integration timing setting isadjusted, as needed, within a predetermined range of integration timingvalues based on processed image characteristics to achieve target imagecharacteristics. If the integration timing setting is at a minimum ormaximum value within the range of available integration timing values,then the gain mode is adjusted to a higher gain mode (integration timingat maximum value) or lower gain mode (integration timing at minimumvalue). Images are then captured and processed using the new integrationand gain settings.

Where applicable, various embodiments provided by the present disclosurecan be implemented using hardware, software, or combinations of hardwareand software. Also where applicable, the various hardware componentsand/or software components set forth herein can be combined intocomposite components comprising software, hardware, and/or both withoutdeparting from the spirit of the present disclosure. Where applicable,the various hardware components and/or software components set forthherein can be separated into sub-components comprising software,hardware, or both without departing from the spirit of the presentdisclosure. In addition, where applicable, it is contemplated thatsoftware components can be implemented as hardware components, andvice-versa.

Software in accordance with the present disclosure, such asnon-transitory instructions, program code, and/or data, can be stored onone or more non-transitory machine readable mediums. It is alsocontemplated that software identified herein can be implemented usingone or more general purpose or specific purpose computers and/orcomputer systems, networked and/or otherwise. Where applicable, theordering of various steps described herein can be changed, combined intocomposite steps, and/or separated into sub-steps to provide featuresdescribed herein.

Embodiments described above illustrate but do not limit the invention.It should also be understood that numerous modifications and variationsare possible in accordance with the principles of the invention.Accordingly, the scope of the invention is defined only by the followingclaims.

What is claimed is:
 1. A system comprising: an image sensor comprising:an array of infrared sensors operable to capture an image of a scene; anintegration timing circuit variably controlling an integration time ofthe infrared sensors; a readout integrated circuit (ROIC) operable togenerate signals from the infrared sensors corresponding to the capturedimage of the scene; an automatic gain controller operable to apply again to the generated signals; and a dynamic range compensationcontroller operable to receive image characteristics corresponding tothe generated signals, compare the received image characteristics totarget image characteristics, and adjust an integration timing signalfor controlling the integration timing circuit to align the receivedimage characteristics with the target image characteristics.
 2. Thesystem of claim 1 wherein the image sensor is a short wave infraredimage sensor.
 3. The system of claim 1 wherein the image characteristicscomprise a received image histogram and a target image histogram andwherein the dynamic range compensation controller is operable to adjustthe integration timing signal to align the received image histogram tothe target image histogram.
 4. The system of claim 3 wherein the dynamicrange compensation controller includes a peak mode wherein the receivedimage histogram is adjusted to align a top of the received imagehistogram with a top of the target histogram.
 5. The system of claim 3wherein the dynamic range compensation controller includes an averagemode wherein the received image histogram is adjusted to align a mean ofthe received image histogram with a mean of the target histogram.
 6. Thesystem of claim 1 wherein the dynamic range compensation controllercomprises an automatic exposure control (AEC) module operable to adjustthe integration timing signal, and a gain control module operable togenerate a gain control signal for controlling the automatic gaincontroller.
 7. The system of claim 6 wherein the AEC module is furtheroperable to detect whether the integration timing signal is equal to orbelow a minimum integration threshold, and wherein the gain controlmodule is operable to adjust the gain control signal to an availablelower gain mode if the image characteristics and the target imagecharacteristic are not aligned.
 8. The system of claim 6 wherein the AECmodule is further operable to detect whether the integration timingsignal is equal to or above a maximum integration threshold, and whereinthe gain control module is operable to adjust the gain control signal toa higher gain mode when the image characteristics and the target imagecharacteristic are not aligned.
 9. The system of claim 1 wherein thedynamic range compensation controller is further operable to calculate astep size based on the comparison between the received imagecharacteristics and the target image characteristics and wherein theintegration timing signal is adjusted by applying the step size to acurrent integration timing signal.
 10. The system of claim 1 wherein thedynamic range compensation controller comprises a state machine operableto perform automatic exposure control.
 11. A method comprising:capturing an image of a scene using an array of infrared sensors;variably controlling an integration time of the infrared sensors usingan integration timing circuit; generating signals from the infraredsensors corresponding to the captured image of the scene using a readoutintegrated circuit (ROTC); applying a gain to the generated signals,using an automatic gain controller; and compensating for a dynamic rangeof the image using a dynamic range controller, including receiving imagecharacteristics corresponding to the generated signals, comparing thereceived image characteristics to target image characteristics, andgenerating an integration timing signal for controlling the integrationtiming.
 12. The method of claim 11 wherein the array of infrared sensorscomprise an array of short wave infrared image sensors.
 13. The methodof claim 11 wherein the image characteristics comprise a received imagehistogram and a target image histogram and wherein the method furthercomprises adjusting the integration timing signal to align the receivedimage histogram to the target image histogram.
 14. The method of claim13 wherein the adjusting the integration timing signal comprisesoperating in a peak mode wherein the received image histogram isadjusted to align a top of the received image histogram with a top ofthe target histogram.
 15. The method of claim 13 wherein the adjustingthe integration timing signal comprises operating in an average modewherein the received image histogram is adjusted to align a mean of thereceived image histogram with a mean of the target histogram.
 16. Themethod of claim 11 further comprising adjusting the integration timingsignal using an automatic exposure control (AEC) module, and generatinga gain control signal for controlling the automatic gain controller. 17.The method of claim 16 further comprising detecting whether theintegration timing signal is equal to or below a minimum integrationthreshold, and adjusting the gain control signal to an available lowergain mode if the received image characteristics and the target imagecharacteristic are not aligned.
 18. The method of claim 16 furthercomprising detecting whether the integration timing signal is equal toor above a maximum integration threshold, and adjusting the gain controlsignal to a higher gain mode when the image characteristics and thetarget image characteristic are not aligned.
 19. The method of claim 11further comprising calculating a step size based on the comparisonbetween the received image characteristics and the target imagecharacteristics and adjusting the integration timing signal by applyingthe step size to a current integration timing signal.
 20. The method ofclaim 11 further comprising operating a state machine to performautomatic exposure control.